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ABSTRACT 

The system of high- stakes accountability in the Kentucky 
public schools raises the question of whether teachers and administrators 
should be held accountable if test scores are influenced by external factors 
over which educators have no control . This study investigates whether such 
external factors , or "contextual effects," bias the accountability index 
scores. The focus is on the Kentucky school district accountability index 
scores for the 1992-94 and 1994-96 biennial periods. District scores were 
chosen as the focus because more suitable measures of contextual effects are 
available. Three contextual effects are considered: (1) median household 

income in the district; (2) teen birth rate; and (3) rural -metropolitan 
differences among districts. Contextual effects might slow or promote a 
school district’s ability to improve its scores, or they might only influence 
the scores within districts. A second set of analyses were done on the 4th, 
8th, and 12th grade scores within each school district. Major findings are 
that contextual effects have little influence on the change in scores within 
districts. This supports the argument of high-stakes accountability 
advocates, who assert that since improvement is measured within the school or 
district, externally imposed advantages or disadvantages will not affect the 
results. Contextual effects have a large influence on differences in scores 
between districts. Between 30 and 40% of the variation in the scores is 
attributable to contextual effects. Median household income is the strongest 
of these effects, with rural-urban differences next. When contextual effects 
are controlled, rural school districts perform better than their 
accountability scores suggest and better than metropolitan districts. Urban 
schools perform less well than their scores suggest when contextual effects 
are controlled. Independent school districts tended to score higher than 
county school districts. Analyses from the individual grades generally 
support the overall analyses. An appendix lists scores and standardized 
? :esidual for each district. (Contains 2 figures, 11 tables, and 20 
-ef erences . ) ( SLD) 
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Executive Summary 



DO CONTEXTUAL EFFECTS BIAS KENTUCKY SCHOOL DISTRICT 
ACCOUNTABILITY INDEX SCORES? 



Kentucky’s system of high-stakes accountability raises the question: Should teachers and 
school administrators be held accountable for student test results if the scores are influenced by 
external factors over which these educators have no control? The goal of the present study is to 
investigate if such external factors, or “contextual effects,” bias the accountability index scores. 
The issue is important because school districts, schools, and educators should be assessed in a fair 
manner. 

The focus of the study is on the Kentucky school district accountability index scores for 
the 1992-94 and 1994-96 biennia. District scores, rather than school scores, were chosen as the 
focus because more suitable measures of contextual effects are available for school districts than 
for schools. Three contextual effects are considered: ( 1 ) median household income in the district, 
(2) teen birth rate, and (3) rural-metropolitan differences among districts. 

Two ways of determining the influence of contextual effects on the test scores are dealt 
with. First, contextual effects might retard or promote a school district’s ability to improve its 
scores. A confirmation of this tendency would pose a serious problem for the current 
accountability system. Second, contextual effects might only influence the differences in scores 
between school districts. This finding would not challenge Kentucky’s current procedure of 
recognizing improvement, which compares the present performance against the previous biennial 
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score within each educational unit. But it could mean that disadvantaged districts and schools will 
lag behind the advantaged ones no matter how much they improve on their benchmarks. This 
becomes a salient issue for the State’s long-range goal of having all schools and districts achieve 
proficiency within twenty years. 

A second set of analyses was done on the fourth, eighth, and twelfth grade scores within 
each school district. These analyses corroborate the results obtained from studying the overall 
district-level scores. Moreover, the grade-level results provide evidence whether contextual 
effects also influence the performance of individual schools. 

The major findings of the study are: 

• Contextual effects have little influence on the change in scores within districts. This 
finding supports the argument of high-stakes accountability advocates who assert that, 
since improvement is measured within a district or school, externally imposed advantages 
or disadvantages will not affect the result. 

• Contextual effects have a large influence on differences in the scores between districts. In 
this instance, between 30 and 40 percent of the variation in the scores is attributable to 
contextual effects. Median household income is the strongest of these effects. Rural- 
metro differences are next strongest, followed by the teen birth rate. These findings warn 
of a danger of very substantial bias when school districts and schools are ranked using 
their accountability scores without first controlling for contextual effects. 

• When the contextual effects are controlled, rural school districts perform better than their 
accountability scores suggest and better than metro districts. Metro school districts, on 
the other hand, perform less well than their scores suggest and less well than their rural 
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counterparts. The reason for these findings may be that rural school districts have less 
bureaucracy to contend with and stronger community support than metropolitan districts. 
However, this advantage of rural districts only becomes evident when the effects of 
median household income and teen birth rate are held constant. 

• On average, the independent school districts score higher than county school districts on 
the accountability tests. After adding the contextual effects on the district scores, it was 
found that all of the positive and negative outliers were independent districts. In other 
words, when district scores are compared statewide and the contextual effects are held 
constant, the highest and lowest performing districts are independents. 

• The analyses of fourth, eighth, and twelfth grade scores in each district generally support 
the results obtained from the overall district-level analyses. The contextual effects are 
found to increase with the grade level. These results suggest that contextual effects will 
also influence cross-school comparisons of scores but not within-school improvement in 
scores. As above, these results bode well for Kentucky’s short-run accountability goals 
but raise a note of caution about whether disadvantaged schools can achieve the long-run 
objective of proficiency. 
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Background 

Recent educational reform initiatives in the United States focus on monitoring and 
improving outcomes and holding school districts and schools accountable for what their students 
learn (Ladd 1996). The current interest in performance-based education reflects in part the desire 
of top management, consumers and taxpayers alike to see organizational efficiency enhanced in 
many arenas— from private business and medical care to the military and education. But purely 
educational concerns also spur the interest in performance-based education. First, there is a 
widespread alarm that American children are not being sufficiently well trained to compete with 
other countries. More effective learning must be promoted to keep the U.S. labor force 
competitive. Second, the costs of education in the United States appear to have grown rapidly 
while what children learn does not show a corresponding increase. 1 Because of this, legislators 
and taxpayers are questioning if they are getting their money’s worth in public education. Third, 
it is argued that the U.S. educational system must become more productive since funds to support 
education could be less available in the future. For many people, increasing the accountability of 
teachers and school systems is felt to be a way to address each of these concerns. 

Currently, there are two main views about how to obtain greater efficiency and 
accountability in educational systems. The first view opts for creating an open market of 
educational services through voucher plans or tax rebates that offset the costs of alternative 
schooling. Parents in the role of consumers can then choose whichever school they believe will 
best educate their child. This perspective relies on competition between schools (within the public 



'Ladd (1996: 2-3) argues that the perception in the United States that the costs of 
education have been increasing sharply is incorrect. 
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sector but also between the public and private sectors) to foster efficiency and academic 
excellence. Advocates say that parents will want their children to attend the better, more efficient 
schools. When such choices are aggregated, schools that are too inefficient to compete for a 
“market share” of students will fail, leaving only the more efficient schools. The second view on 
attaining more effective schools relies on an administered system of accountability that recognizes 
and rewards success in the public schools. This perspective has the advantage of the first because 
it keeps the present administrative structure of public school systems largely intact while 
reforming that structure to achieve accountability goals. Realistically speaking, an administered 
accountability system may be the only choice for states like Kentucky that have many rural school 
districts. Because of the higher cost of delivering education to rural areas, the open- 
mar ket/parental-cho ice approach seems better suited for metropolitan areas. 

Over the past decade these same forces have set the stage for a radical change in 
Kentucky's educational system, but concerns more specific to the State provided the immediate 
motivation for reform. Kentucky students had a record of performing below the national average 
on achievement tests, and high school graduation rates were low. The catalyst for reform was a 
law suit in which the Kentucky Supreme Court upheld a lower court decision that the State’s 
educational system was unconstitutional. Public schools were not being funded equitably, and 
Kentucky students did not have equal educational opportunities. 

The Kentucky Education Reform Act of 1990 (KERA) launched one of the nation’s most 
ambitious educational reform efforts of recent decades (Elmore, Abelmann, and Fuhrman 1996). 
One of KERA's most noteworthy accomplishments has been the high-stakes performance 
assessment program, called the Kentucky Instructional Results Information System (KIRIS). This 



program embraced a complex procedure for testing students using authentic assessment methods. 
The results from the assessments have become the basis for granting monetary rewards to schools 
that show significant improvement and to levy sanctions against those that fail to show progress 
(Guskey 1994). Controversial from the start, attacks on KIRIS have intensified recently and may 
lead to substantial changes in the accountability program. Still, the demand in Kentucky for an 
effective means of assessing and improving educational performance will not go away. Whatever 
the shortcomings, KIRIS has put educators on notice that improved performance is expected. 

The Present Study 

The design and implementation of an administered accountability system raise many thorny 
questions (Coleman 1995; Ladd 1996): Who should be held accountable? Teachers and school 
officials primarily, or should parents and students be held accountable too? How will educational 
outcomes be measured? With criterion-referenced tests or with more “authentic” procedures such 
as writing portfolios and group problem-solving activities, even if the latter methods are expensive 
to administer and score? Whatever the method of assessment, how will problems such as 
“teaching to the test” and cheating be minimized? What is an appropriate way to link 
performance to rewards and sanctions? Finally, can performance be measured in a way that is fair 
to all school systems despite strong disparities of wealth, community mores, and geographic 
location? 

The present study addresses only the last of these questions. It examines the Kentucky 
accountability index scores for public school districts during two biennia, 1992-94 and 1994-96. 
The purpose of the analysis is to find out the extent to which external contextual effects bias the 
scores and thus detract from their usefulness as measures for accountability. The issue is that 
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teachers and school administrators should not be held accountable for socioeconomic factors that 
influence student performance but are beyond the control of school personnel. If such contextual 
effects are discovered to bias the accountability scores, the system is unfair. 

Research on Contextual Effects and Educational Outcomes 
In this study, the term “contextual effects” refers to factors in the social environment of a 
school district which influence student performance on the accountability test. Contextual effects 
are external to the schools and are not subject to the direct control of teachers and administrators. 
Studies showing that contextual effects do indeed influence educational outcomes are not new. 
Equality of Educational Opportunity (Coleman et al. 1966) reached the (at the time surprising and 
highly controversial) conclusion that much variation in student performance was not the result of 
differences between schools but was due instead to the disparate family and community 
backgrounds of students. Since this watershed study, many studies (for instance, Coleman 1988, 
1990; Hallinan 1988; Schneider and Coleman 1993; Smith, Beaulieu, and Israel 1992; Teachman, 
Carver, and Paasch 1 997) have reaffirmed the importance of contextual effects. 2 Many of these 
studies have employed large national probability samples of students such as the High School and 
Beyond Database and the National Educational Longitudinal Study Database. One pre-KERA 
study in Kentucky noted the effect of economic deprivation on school district performance 
(Guskey and Kifer 1990). To date, most of the research has focused on how student achievement 
varies with the family’s socioeconomic circumstances and social capital resources. 3 The issue of 



2 Eflfective schools research commonly employs family SES indicators to capture external 
factors that influence learning (Hanushek 1997; Lee, Smith, and Croninger 1997; Zigarelli 1996). 

3 James Coleman (1988, 1990) introduced the idea of “social capital” to embrace a set of 
factors that must be considered when explaining student outcomes. These factors included 
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